Stochastic first-order methods for convex and nonconvex functional constrained optimization

Functional constrained optimization is becoming more and important in machine learning operations research. Such problems have potential applications risk-averse learning, semisupervised robust among others. In this paper, we first present a novel Constraint Extrapolation (ConEx) method for solving convex functional problems, which utilizes linear approximations of the constraint functions to define extrapolation (or acceleration) step. We show that unified algorithm achieves best-known rate convergence different composite including or strongly convex, smooth nonsmooth with stochastic objective and/or constraints. Many these rates were fact obtained time literature. addition, ConEx single-loop does not involve any penalty subproblems. Contrary existing primal-dual methods, it require projection Lagrangian multipliers into (possibly unknown) bounded set. Second, nonconvex introduce new proximal point transforms initial problem sequence by adding quadratic terms both Under certain MFCQ-type assumption, establish KKT points when subproblems are solved exactly inexactly. For large-scale practical approximate solutions computed aforementioned method. strong feasibility total iteration complexity required inexact variety settings, To best our knowledge, most results also seem be